Image processing apparatus and image processing method including area setting and perspective conversion

ABSTRACT

An image processing apparatus and method can combine images, while maintaining continuity without causing discomfort, that capture the surrounding area of an automobile. An image processing apparatus includes an image acquisition unit that acquires first and second images respectively capturing surrounding areas of an automobile including first and second areas, an outline detector that performs outline detection on the first and second images and generates first and second outlines respectively, a determiner that determines whether the first and second outlines each include an outline of the same object, and a area selector that performs area setting and perspective conversion on the first or second image when the first and second outlines are determined to include an outline of the same object, so that in a combined image generated by combining at least the first and second images, the first and second outlines are continuous at the same object.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit of Japanese PatentApplication No. 2014-036067 filed Feb. 26, 2014 and PCT Application No.PCT/JP2015/001015, the entire contents of which are incorporated hereinby reference.

TECHNICAL FIELD

This disclosure relates to an image processing apparatus and imageprocessing method that accurately combine a plurality of images thatcapture the surrounding area of a vehicle, without causing any sense ofdiscomfort.

BACKGROUND

Systems for assisting with the driving of a vehicle, such as anautomobile, have been proposed. For example, one known system generatesa 360° overhead image by capturing the vehicle periphery with aplurality of on-board cameras and combining the images (overhead imagegeneration system). The overhead image generation system for exampleassists with driving when parking by showing the driver the combinedimage.

The on-board cameras used in such a system are mounted by adjusting theposition and orientation when installing the cameras on the automobile,so that no discontinuity occurs at the seams between images. For reasonssuch as vibration of the automobile or an external shock, however, theposition and orientation of attachment might end up shifting. In such acase, continuity is lost at the image seams in the combined image.

JP 2010-166196 A (PTL 1) discloses an overhead image generation systemin which distant 360° circular images are combined to suppress theoccurrence of overlap at the seams between images of the cameras.

CITATION LIST Patent Literature

PTL 1: JP 2010-166196 A

SUMMARY Technical Problem

By including distant 360° circular images, however, the method disclosedin PTL 1 reduces the area that becomes the seam in a nearby front image,right image, back image, and left image. In other words, PTL 1 preventsa discontinuous portion at the seams in near images from beingconspicuous but does not correct the discontinuous portion. Accordingly,even if the method in PTL 1 is used, the driver may feel a sense ofdiscomfort at the seams of the combined image.

Therefore, it would be helpful to provide an image processing apparatusand image processing method that can combine a plurality of images,while maintaining continuity without causing any sense of discomfort,that capture the surrounding area of a vehicle.

Solution to Problem

In order to resolve the aforementioned problem, an image processingapparatus of this disclosure includes:

an image acquisition unit configured to acquire a first image capturinga surrounding area of an automobile including a first area and a secondimage capturing a surrounding area of the automobile including a secondarea adjacent to the first area;

an outline detector configured to perform outline detection on the firstimage and the second image and to generate a first outline and a secondoutline respectively;

a determiner configured to determine whether the first outline and thesecond outline each include an outline of a same object; and

a area selector configured to perform area setting and perspectiveconversion on the first image or the second image when the determinerdetermines that the first outline and the second outline include anoutline of the object, so that in a combined image generated bycombining at least the first image and the second image, the firstoutline and the second outline are continuous at the object.

In order to resolve the aforementioned problem, an image processingapparatus of this disclosure includes:

an image acquisition unit configured to acquire a first image capturinga surrounding area of an automobile including a first area and a secondimage capturing a surrounding area of the automobile including a secondarea adjacent to the first area; and

a area selector configured to perform area setting and perspectiveconversion on the first image or the second image when a same objectappears in each of the first image and the second image due to movementof the automobile, so that a position and an orientation of the objectin one of the first image and the second image match a position and anorientation sought by calculation based on a position and an orientationof the object in the other one of the first image and the second image.

In order to resolve the aforementioned problem, an image processingmethod of this disclosure includes:

acquiring a first image capturing a surrounding area of an automobileincluding a first area and a second image capturing a surrounding areaof the automobile including a second area adjacent to the first area;

performing outline detection on the first image and the second image andgenerating a first outline and a second outline respectively;

determining whether the first outline and the second outline eachinclude an outline of a same object; and

performing area setting and perspective conversion on the first image orthe second image when the first outline and the second outline aredetermined to include an outline of the object, so that in a combinedimage generated by combining at least the first image and the secondimage, the first outline and the second outline are continuous at theobject.

In order to resolve the aforementioned problem, an image processingmethod of this disclosure includes:

acquiring a first image capturing a surrounding area of an automobileincluding a first area and a second image capturing a surrounding areaof the automobile including a second area adjacent to the first area;and

performing area setting and perspective conversion on the first image orthe second image when a same object appears in each of the first imageand the second image due to movement of the automobile, so that aposition and an orientation of the object in one of the first image andthe second image match a position and an orientation sought bycalculation based on a position and an orientation of the object in theother one of the first image and the second image.

Advantageous Effect

An image processing apparatus and image processing method according tothis disclosure can combine a plurality of images, while maintainingcontinuity without causing any sense of discomfort, that capture thesurrounding area of a vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

In the accompanying drawings:

FIG. 1 is a block diagram schematically illustrating the structure of animage processing apparatus according to an embodiment and a camerasystem provided with the image processing apparatus;

FIG. 2 schematically illustrates the arrangement of constituent elementsof the camera system in FIG. 1;

FIG. 3 illustrates an example of a combined image generated by the imageprocessing apparatus of FIG. 1;

FIG. 4 illustrates an example of a combined image to be corrected;

FIG. 5 is a flowchart illustrating operations by the image processingapparatus of Embodiment 1;

FIG. 6 illustrates an example of a combined image to be corrected;

FIG. 7 is a flowchart illustrating operations by the image processingapparatus of Embodiment 2; and

FIG. 8 illustrates an example of a combined image to be corrected.

DETAILED DESCRIPTION Embodiment 1

The following describes Embodiment 1 with reference to the drawings.

First, an image processing apparatus 12 and a camera system 10 accordingto Embodiment 1 are described. FIG. 1 is a functional block diagramschematically illustrating the structure of the camera system 10, whichis provided with the image processing apparatus 12.

As illustrated in FIG. 1, the camera system 10 is provided with aplurality of image pickup apparatuses (front camera 11 a, rear camera 11b, left-side camera 11 c, and right-side camera 11 d), the imageprocessing apparatus 12, and a display apparatus 13. In this embodiment,each of the constituent elements of the camera system 10 can transmitand receive information via a network 14 such as a dedicated line or aController Area Network (CAN). In the camera system 10, the imageprocessing apparatus 12 may be provided with a display having the samefunctions as the display apparatus 13.

As illustrated in FIG. 2, the front camera 11 a is positioned so as tobe able to capture the surrounding area at the front of a vehicle 15,such as an automobile. The rear camera 11 b is positioned so as to beable to capture the surrounding area at the back of the vehicle 15. Theleft-side camera 11 c and the right-side camera 11 d are for examplepositioned to face vertically downward at the left and right doormirrors 16 so as to be able to capture the surrounding area at the sidesof the vehicle 15. In FIG. 2, the right-side camera 11 d is hidden bythe body of the vehicle 15 and therefore is not illustrated. Theleft-side camera 11 c and the right-side camera 11 d are symmetricallydisposed respectively on the left and right sides of the vehicle 15. Thedisplay apparatus 13 is disposed at a position visible from the driver'sseat.

The front camera 11 a, rear camera 11 b, left-side camera 11 c, andright-side camera 11 d are provided with a lens having a wide-angleview, such as a fisheye lens, and can take wide-angle shots of thesurrounding area of the vehicle 15. Typically, objects can bephotographed over a wide range with wide-angle photography, and objectsin the image periphery are curved. The curvature, however, is correctedby the image pickup apparatus itself or by the image processingapparatus 12 and then displayed on the display apparatus 13. Thecaptured images from the front camera 11 a, rear camera 11 b, left-sidecamera 11 c, and right-side camera 11 d in this embodiment respectivelycorrespond to the front area Ai, rear area Bi, left-side area Ci, andright-side area Di of the vehicle 15 as illustrated in FIGS. 3, 4, 6,and 8.

Next, referring again to FIG. 1, the structure of the front camera 11 a,rear camera 11 b, left-side camera 11 c, and right-side camera 11 d isdescribed. The front camera 11 a is provided with an optical system 17a, an image pickup element 18 a, an image processor 19 a, and cameracontroller 20 a.

The optical system 17 a is configured to include a diaphragm and a lensand forms an image of an object. In this embodiment, the optical system17 a has a wide-angle view, and as described above, can capture objectsincluded in the surrounding area of the vehicle 15.

The image pickup element 18 a may, for example, be a CMOS image pickupelement and picks up the object image formed by the optical system 17 a.The image pickup element 18 a also outputs the captured image generatedby image pickup to the image processor 19 a as an image signal.

The image processor 19 a may, for example, be a dedicated processor forimage processing, such as a DSP. The image processor 19 a appliespredetermined image processing to the image signal acquired from theimage pickup element 18 a, such as noise removal, color interpolation,brightness correction, color correction, gamma correction, and whitebalance. The image processor 19 a outputs the image signal to whichregular image processing has been applied to the image processingapparatus 12.

The camera controller 20 a may, for example, be a dedicatedmicroprocessor or a general purpose CPU that executes specific functionsby reading a specific program. The camera controller 20 a controlsoperations of each portion of the front camera 11 a. For example, thecamera controller 20 a controls operations of the image pickup element18 a and the image processor 19 a and periodically outputs an imagesignal, for example every 30 fps.

Like the front camera 11 a, the rear camera 11 b, left-side camera 11 c,and right-side camera 11 d are respectively provided with opticalsystems 17 b, 17 c, and 17 d, image pickup elements 18 b, 18 c, and 18d, image processors 19 b, 19 c, and 19 d, and camera controllers 20 b,20 c, and 20 d. The functions and structure of the optical systems 17 b,17 c, and 17 d, image pickup elements 18 b, 18 c, and 18 d, imageprocessors 19 b, 19 c, and 19 d, and camera controllers 20 b, 20 c, and20 d are similar to those of the front camera 11 a.

The image processing apparatus 12 is provided with an image acquisitionunit 21, a control information acquisition unit 22, an outline detector27, a sameness determiner 28, and a combined area selector 29. Each ofthe functional units of the image processing apparatus 12 may, forexample, be implemented with a dedicated microprocessor or a generalpurpose CPU that executes specific functions by reading a specificprogram.

Via the network 14, the image acquisition unit 21 acquires the capturedimages of the surrounding area in each corresponding direction from thefront camera 11 a, rear camera 11 b, left-side camera 11 c, andright-side camera 11 d. The image acquisition unit 21 may acquire all oronly a portion of the captured images of the surrounding areas to thefront, rear, and sides (right and left). As described below, in theimage processing apparatus 12 of this embodiment, processing such asrotation or reduction might be performed on the captured images of thesurrounding areas, and the image acquisition unit 21 thereforepreferably acquires captured images at a larger pixel size than capturedimages for a regular combined image.

The control information acquisition unit 22 acquires control informationon the vehicle 15. The control information is a variety of informationrelated to the state of the vehicle 15, for example informationindicating forward or backward movement, steering angle, and speed ofthe vehicle 15. The control information acquisition unit 22 can acquirethe control information by any method, such as acquisition of thecontrol information from the vehicle 15 over the network 14 oracquisition, over a wired or wireless connection, of control informationoutput by another constituent element provided in the vehicle 15.

As described below, the combined area selector 29 generates a combinedimage using a plurality of captured images acquired by the imageacquisition unit 21. In this embodiment, the combined image is a 360°overhead image of the vehicle 15. An overhead image refers to an imageof the surrounding area of the vehicle 15 when viewing the vehicle 15vertically downward from above.

First, the combined area selector 29 cuts out a predetermined rangecorresponding to the size of the combined image from each captured imageused to generate the combined image. The combined area selector 29applies processing for perspective conversion to the plurality ofcaptured images that were cut out to convert the images to overheadimages.

Next, the combined area selector 29 generates a combined image using theplurality of captured images to which processing for perspectiveconversion was applied. Furthermore, to generate the combined image, thecombined area selector 29 also uses an image of the vehicle 15 viewedvertically downward from above. For example, in the combined imageillustrated in FIG. 3, an image indicating the vehicle 15 is used in apartial area (vehicle display area) Ei at the center of the combinedimage. The image of the front camera 11 a is used in the front area Aiin front of the area Ei. The image of the rear camera 11 b is used inthe rear area Bi behind the area Ei. The image of the left-side camera11 c is used in the left-side area Ci to the left of the area Ei. Theimage of the right-side camera 11 d is used in the right-side area Di tothe right of the area Ei.

FIG. 4 illustrates a white line WL (see FIG. 3) on the road surface inthe combined image being discontinuous at the borders B2 and B3 as aresult of a shift in the image pickup range due to the attachmentposition of the left-side camera 11 c being shifted by, for example, anexternal shock. If the generated combined image for example has adiscontinuity as in FIG. 4, then in accordance with information from theoutline detector 27 or the like, the combined area selector 29 executesprocessing for setting the area and for perspective conversion of aspecific captured image and generates a new combined image that hascontinuity. Details on the area setting and the perspective conversionare provided below, but in short, area setting refers to setting theratio of expansion/reduction and the shift amount.

The outline detector 27 extracts the outline of two adjacent capturedimages. The outline detector 27 may use a known method to extract theoutline, such as a method to calculate the difference between adjacentpixels, a method that uses a Sobel filter, or a method that uses aLaplacian filter. The images of outlines (outline images) extracted bythe outline detector 27 in correspondence with two adjacent capturedimages are output to the sameness determiner 28 and the combined areaselector 29 and are used to determine the continuity of the outlineimages, and for area setting and perspective conversion of the capturedimages. While details are provided below, it suffices for the outlinedetector 27 to extract the outline of an object extending across theborder between two adjacent captured images when the captured images arecombined or to extract the outline of an object existing near theborder. In other words, the outline detector 27 does not need to extractthe outline of all of the objects in two adjacent captured images.

Based on the outline images of two adjacent captured images, thesameness determiner 28 determines whether objects appearing in the twocaptured images include the same object. Examples of the same objectappearing in two captured images include a display on the road surface(a white line of a side strip or center line, or a display of a speedlimit or an indication to stop), a guardrail, a side ditch, a roadshoulder (sidewalk), or the like. The vehicle 15 may be moving or may bestopped. The sameness determiner 28 may use a Hough transform to detectthe same object correctly. By using a Hough transform, an object can bedetected correctly even if the outline of the object is discontinuous inthe outline image.

When the sameness determiner 28 determines that the objects are thesame, the result of determination is communicated to the combined areaselector 29. So that the outlines of the same object in the two adjacentcaptured images are continuous in the combined image of the capturedimages, the combined area selector 29 performs area setting andperspective conversion setting of the captured images and can thusgenerate a combined image with no discontinuity.

The display apparatus 13 may, for example, be configured to include anLCD and can display a moving image in real time. The display apparatus13 displays the combined image output by the image processing apparatus12 over the network 14. The display apparatus 13 may, for example, beconfigured as a touch panel and may also function as an interface toaccept user operation.

With reference to the drawings, an example of the combined imageincluding a discontinuity is provided below, and processing by the imageprocessing apparatus 12 is then described.

For comparison, with reference to FIG. 3, an example is first providedof an image displayed on the display apparatus 13 for the case of aplurality of images that capture the surrounding area of the vehicle 15being combined accurately and without causing any sense of discomfort.The combined area selector 29 cuts out and combines the captured imagesof the front area Ai, rear area Bi, left-side area Ci, and right-sidearea Di of the vehicle 15 and displays, on the display apparatus 13, acombined image that is a 360° overhead image of the vehicle 15.

In the example in FIG. 3, the surface of the road on which the vehicle15 is driven, a white line WL (such as a side strip) on the roadsurface, and a white line WR (such as a center line) are displayed onthe display apparatus 13. The borders B0, B1, B2, and B3 respectivelyindicate the borders between captured images at the front area Ai andthe right-side area Di, the right-side area Di and the rear area Bi, therear area Bi and the left-side area Ci, and the left-side area Ci andthe front area Ai. In the example in FIG. 3, the white line WL exhibitsno discontinuity at the borders B2 and B3, and the white line WR alsoexhibits no discontinuity at the borders B0 and B1. In other words,since the white line WL and the white line WR neither differ inthickness nor have any discontinuity, the plurality of images capturingthe area surrounding the vehicle 15 are combined accurately and withoutcausing any sense of discomfort. The xy-axes in FIG. 3 are virtualcoordinate axes for the combined image. In the example in FIG. 3, thevehicle 15 moves forward in the direction of the y-axis, and the whiteline WL and the white line WR extend in the direction of the y-axis.

On the other hand, as already described above, the example in FIG. 4illustrates a discontinuity in the white line WL at the borders B2 andB3. Such a discontinuity may, for example, occur due to the position ofattachment or the orientation of the image pickup apparatus shifting asa result of vibration of the vehicle 15 during driving, an externalshock, or other such reason. With the processing described below, theimage processing apparatus 12 in this embodiment detects and correctsthe discontinuity occurring at the borders of a combined image such asthe one illustrated in FIG. 4 and displays an accurate combined imagehaving continuity, such as the image illustrated in FIG. 3, on thedisplay apparatus 13.

The flow of processing by the image processing apparatus 12 is describedwith reference to FIG. 5. Each step is described in detail using theexample in FIG. 4. An outline of the processing is as follows. The imageacquisition unit 21 acquires adjacent captured images and outputs thecaptured images to the outline detector 27 and the combined areaselector 29. The outline detector 27 extracts outlines from the receivedadjacent captured images and outputs the extracted outlines to thesameness determiner 28 and the combined area selector 29. The samenessdeterminer 28 determines whether the outlines extracted at the borderbetween adjacent captured images include the same object (for example,the white line of a side strip or center line) and outputs thedetermination result to the combined area selector 29. When receiving aresult from the sameness determiner 28 indicating that the same objectis included, the combined area selector 29 performs area setting andperspective conversion on one of the captured images and then generatesa combined image, so that the outlines extracted at the border betweenadjacent captured images is not discontinuous. Details are providedbelow.

The image acquisition unit 21 of the image processing apparatus 12acquires a first image and a second image (step S2). Among the capturedimages of the front area Ai, rear area Bi, left-side area Ci, andright-side area Di, the first image and the second image are twoadjacent captured images. The image acquisition unit 21 for exampleoutputs the captured image of the front area Ai as the first image andthe captured image of the left-side area Ci as the second image to theoutline detector 27 and the combined area selector 29. In this example,the front area Ai and the left-side area Ci respectively correspond tothe first area and the second area of this disclosure. The processingdescribed below is executed by the outline detector 27, the samenessdeterminer 28, and the combined area selector 29 on the captured imageof the front area Ai and the captured image of the left-side area Ci.Subsequently, the image acquisition unit 21 for example outputs thecaptured image of the left-side area Ci as the first image and thecaptured image of the rear area Bi as the second image to the outlinedetector 27 and the like. In this way, the image acquisition unit 21outputs a pair of adjacent captured images that capture the areasurrounding the vehicle 15 to the outline detector 27 and the like whilechanging the combination of captured images.

The outline detector 27 of the image processing apparatus 12 performsoutline detection on each image in the received pair of adjacentcaptured images (first image and second image) and generates a firstoutline and a second outline (step S4).

The outline detector 27 for example uses the aforementioned Laplacianfilter or the like to perform the outline detection. The first outlineand the second outline are outlines detected respectively for the firstimage and second image. In this embodiment, the first outline and thesecond outline do not include the outlines of all of the objects in thefirst image and the second image. The first outline and the secondoutline are used by the sameness determiner 28 and the combined areaselector 29 so as to generate a combined image with no discontinuity atthe border when the first image and the second image are combined.Therefore, it suffices for the first outline and the second outline toextract the outline of an object extending across the border when thefirst image and the second image are combined or to extract the outlineof an object existing near the border.

For example, in FIG. 4, suppose that the captured image of the frontarea Ai is the first image, and the captured image of the left-side areaCi is the second image. In this case, the first outline includes theoutline of a portion of the extracted white line WL (white line WLa) butdoes not include the outline of a portion of the white line WR. Thereason is that the white line WR is distant from the border between thefirst image and the second image and cannot be considered to be near theborder. The second outline includes the outline of a portion of theextracted white line WL (white line WLc). The outline detector 27outputs an image including the first outline (first outline image) andan image including the second outline (second outline image) to thesameness determiner 28.

The sameness determiner 28 of the image processing apparatus 12 receivesthe first outline image and the second outline image. The samenessdeterminer 28 then determines whether the first outline and the secondoutline are of the same object (step S6). Step S6 corresponds to thestep of determining sameness in this disclosure.

The sameness determiner 28 can determine that the first outline and thesecond outline are outlines of the same object by estimating that thefirst outline and the second outline are outlines of, for example, adisplay on the road surface, a guardrail, a side ditch, a road shoulder,or the like based, for example, on characteristics of the first outlineand second outline. For example, a side strip, center line, or otherdisplay on the road surface, a guardrail, a side ditch, a road shoulder,or the like have the characteristics of being long and parallel to theroad on which the vehicle 15 is being driven.

Whereas a white line on the road surface, such as a side strip, and aroad shoulder are straight lines that extend over a long distancewithout interruption, a center line, side ditch, guardrail, or the likemay be a broken line. At this point, by estimating a straight line usinga Hough transform, the sameness determiner 28 can improve the accuracyof the determination that an object is a display on the road surface orthe like. The sameness determiner 28 determines whether the firstoutline and the second outline are outlines of the same object, such asa display on the road surface, and outputs the determination result tothe combined area selector 29. The first outline and the second outlineto which the Hough transform has been applied are also output to thecombined area selector 29. In the example in FIG. 4, the entire firstoutline and second outline are outlines of the same object (white lineWL). The first outline and second outline do not, however, entirely needto be outlines of the same object and may be determined by the samenessdeterminer 28 to be outlines of the same object by virtue of simplyincluding outlines of the same object.

When the sameness determiner 28 determines that the first outline andthe second outline are of the same object (step S6: Yes), the combinedarea selector 29 of the image processing apparatus 12 determines whetherthe first outline and the second outline are continuous when combined(step S10). Stating that the first outline and the second outline arecontinuous does not necessarily require that the entire first outlineand second outline be continuous. Rather, continuity in the outlines ofthe objects determined by the sameness determiner 28 to be outlines ofthe same object is sufficient. If the first outline and the secondoutline are not continuous (step S10: No), the combined area selector 29makes corrections so that the first outline and the second outline arecontinuous by setting the area of and performing perspective conversionon either the first image or the second image (step S12).

When the first outline and the second outline are of the same object,the combined area selector 29 determines whether the first outline andthe second outline are connected linearly at the border based on thefirst outline and the second outline received from the samenessdeterminer 28. If the first outline and second outline are a portion ofthe same object, these outlines are represented as straight lines by theaforementioned Hough transform or the like. Therefore, it can bedetermined whether the first outline and the second outline areconnected linearly based on the position and angle of these lines.

At this time, xy coordinates such as those illustrated in FIG. 4 may beused, with the direction of travel of the automobile being one of theaxes. For example, when the white line WLa is parallel to the y-axis andthe white line WLc is inclined, the combined area selector 29 candetermine that the two lines have a discontinuity at the border. At thistime, since the white line WLc is inclined, the combined area selector29 performs correction by perspective conversion on the captured imageof the left-side area Ci so that the white line WLc becomes parallel tothe y-axis. Also, when the width of the outline of the white line WLaand the width of the outline of the white line WLc differ, the combinedarea selector 29 can determine that the two lines have a discontinuityat the border. At this time, the combined area selector 29 sets the areaof the captured image of the left-side area Ci (sets the ratio ofexpansion or reduction and sets the shift amount) so that the width ofthe outline of the white line WLc becomes the same as the width of theoutline of the white line WLa. When the vehicle 15 is moving, thecombined area selector 29 preferably performs correction at the momentwhen the vehicle 15 is moving in a straight line parallel to one of theaxes of the xy coordinate system, based on control information acquiredby the control information acquisition unit 22. At this time, itsuffices for the outline of the same object to be corrected so as to beparallel to one of the axes in the xy coordinate system. Hence, theamount of calculation can be reduced.

After step S12, the combined area selector 29 again determines whetherthe first outline and the second outline are continuous when combined(step S10). When the outlines are continuous (step S10: Yes), thecombined area selector 29 combines the first image and the second image(step S18).

In this way, after step S18, the image acquisition unit 21 outputs apair of adjacent captured images that capture the area surrounding thevehicle 15 to the outline detector 27 and the like while changing thecombination of captured images. Therefore, the outline detector 27,sameness determiner 28, and combined area selector 29 also execute theprocessing sequence for another pair of adjacent surrounding areas. Atthis time, if the combined area selector 29 has performed correction byperspective conversion or the like, the corrected captured image ispreferably output from the image acquisition unit 21 to the outlinedetector 27 and the like in order to avoid duplicate processing.

In step S6, when the sameness determiner 28 determines that the firstoutline and the second outline are not of the same object (step S6: No),the combined area selector 29 terminates processing without performingcorrection.

In the above-described way, the image processing apparatus 12 accordingto this embodiment, which implements the aforementioned image processingmethod, determines whether there is a discontinuity at the borders of aplurality of images that capture the surrounding area of the vehicle 15by extracting the outlines of the same object at or near the borderbetween two adjacent captured images. When there is a discontinuity,correction is performed so that the objects are continuous at theborder, allowing generation of a combined image that maintainscontinuity without causing the viewer any sense of discomfort. Automaticcalibration that adjusts the area setting and perspective conversionsetting of the captured images is thus possible.

Embodiment 2

The following describes Embodiment 2 with reference to the drawings.

Unlike the image processing apparatus 12 according to Embodiment 1, theimage processing apparatus 12 according to this embodiment allowsautomatic calibration that adjusts the area setting and perspectiveconversion setting of the captured images even when the same object isnot included simultaneously in adjacent captured images. The schematicstructure of the image processing apparatus 12 and the camera system 10provided with the image processing apparatus 12 is the same as inEmbodiment 1, and therefore a description of components other than thesameness determiner 28 is omitted (see FIG. 1). In order to avoidredundant description, the following describes the differences from theimage processing apparatus 12 according to Embodiment 1.

In this embodiment as well, based on the outline images of two adjacentcaptured images, the sameness determiner 28 determines whether objectsappearing in the two captured images are the same object. The outlineimages of the two adjacent captured images received by the samenessdeterminer 28, however, are not images captured at the same time butrather are captured with a time difference therebetween. For example,when a delineator P on the road surface first appearing in the capturedimage at the front of the vehicle 15 appears in the captured image atthe side of the vehicle after a predetermined length of time elapses,the sameness determiner 28 receives these captured images and determineswhether the delineators P are the same. When the sameness determiner 28determines that the objects are the same, the result of determination iscommunicated to the combined area selector 29. Based on information onthe position and orientation of the same object in the two capturedimages, the combined area selector 29 performs area setting andperspective conversion setting of the captured images and can thusgenerate a combined image with no discontinuity.

FIG. 6 illustrates the trajectories VLa, VLc, and VLb of the delineatorP on the road surface in the combined image being discontinuous at theborders B2 and B3 as a result of a shift in the image pickup range dueto the attachment position of the left-side camera 11 c being shiftedby, for example, an external shock. Unlike the case of FIG. 4, thedelineator P appears in the combined image, rather than the white lineWL (which is an example of a continuous object). Here, the trajectoriesVLa, VLc, and VLb in FIG. 6 represent virtual trajectories of thedelineator P as a result of the vehicle 15 moving. As illustrated inFIG. 6, in the captured image in the left-side camera 11 c, thedelineator P seems to be moving in a direction at an inclination fromthe direction of travel of the vehicle 15 (y-axis direction), whichcauses a sense of discomfort for the viewer. Accordingly, correction isnecessary as in the example in FIG. 4.

Here, the delineator P is a post, installed along the road, to which areflector is attached. As compared to a guardrail, for example, thedelineator P is small and does not easily appear in two captured imagesat the same time. Therefore, there are cases in which the same method asin Embodiment 1 cannot be used. To address this issue, the imageprocessing apparatus 12 according to this embodiment uses two capturedimages with a time difference to cause the same object (delineator P) toappear in two adjacent captured images, as in FIG. 6. In the example inFIG. 6, the captured image of the left-side area Ci at time t1 and thecaptured image of the front area Ai at the time t0 before the time t1are used. The time difference between the captured images (t1−t0) is setso that due to the movement distance yielded by multiplying the timedifference by the travel speed of the delineator P, the delineator Pmoves from inside the captured area of the front area Ai to inside thecaptured area of the left-side area Ci. The travel speed of thedelineator P may, for example, be obtained from the speed of the vehicle15 acquired by the control information acquisition unit 22.

In the example in FIG. 6, the captured image of the front area Ai(corresponding to the first image) includes the delineator P at time t0,and the captured image of the left-side area Ci (corresponding to thesecond image) includes the delineator P at time t1. These delineators Pare the same object. The following describes the flow of processing bythe image processing apparatus 12 with reference to FIG. 7. Each step isdescribed in detail using the example in FIG. 6.

An outline of the processing is as follows. The image processingapparatus 12 waits for the vehicle 15 to travel straight before startingprocessing. Upon the vehicle 15 traveling straight, the imageacquisition unit 21 acquires adjacent captured images with differentcaptured times and outputs the result to the combined area selector 29.The combined area selector 29 determines whether the same object appearsin the adjacent captured images, and if so, seeks the position andorientation of the object in the chronologically later captured image bycalculation based on the chronologically earlier captured image and thespeed and direction of the vehicle 15. The combined area selector 29then sets the area of, and performs perspective conversion on, thechronologically later captured image and generates a combined image, sothat the position and orientation of the object in the chronologicallylater captured image match the position and orientation sought by theaforementioned calculation. Details are provided below.

The processing by the image processing apparatus 12 in this embodimentrequires that the same objects appear in two adjacent captured images asa result of the movement of the vehicle 15. Therefore, the vehicle 15needs to be moving. Based on the control information acquired by thecontrol information acquisition unit 22, the image processing apparatus12 acquires the movement direction and the speed of the vehicle 15 (stepS20). If the vehicle 15 is not moving straight (step S22: No), the imageprocessing apparatus 12 returns to step S20, whereas when the vehicle 15is moving straight, the image processing apparatus 12 executes thefollowing processing (step S22: Yes). Here, the straight direction ofthe vehicle 15 is preferably parallel to one of the axes in the virtualxy coordinate system of the combined image. As described above, this isbecause the amount of calculation when performing correction can bereduced. Below, the vehicle 15 is described as moving straight in thedirection of the y-axis.

The image acquisition unit 21 of the image processing apparatus 12acquires a first image and a second image (step S24). Since step S24corresponds to step S2 in Embodiment 1, details thereof are omitted.

The combined area selector 29 of the image processing apparatus 12determines whether the same object appears in the first image and thesecond image (step S26). When the same object appears as in the examplein FIG. 6 (step S26: Yes), the movement direction and speed of thevehicle 15 are used to seek the position and orientation of the objectin the second image by calculation based on the position and orientationof the object in the first image (step S30).

When determining whether the same object appears, the combined areaselector 29 may make the determination of sameness by extracting andcomparing feature points. A known method may be used to extract featurepoints, such as a method to extract the luminance difference, cornerdetection, or the like. Based on the position and orientation of thedelineator P in the image with the earlier captured time (the firstimage in this example, i.e. the captured image of the front area Ai),the combined area selector 29 can seek the position and orientation ofthe delineator P in the other image (the second image in this example,i.e. the captured image of the left-side area Ci) by calculation.

First, the combined area selector 29 recognizes that the vehicle 15 istraveling straight in the y-axis direction based on control informationacquired by the control information acquisition unit 22 and calculates aunit vector Vp indicating the movement of the object per unit time inFIG. 6. Since the position of the delineator P at time t0 is P(t0), theposition of the delineator P at time t1, i.e. P(t1), can be sought withthe following equation.P(t1)=Vpx(t1−t0)+P(t0)P(t1) corresponds to the aforementioned position of the delineator Psought by calculation. Since the vehicle 15 is traveling straight in they-axis direction, the direction of the delineator P at time t1 is theopposite direction from the y-axis, and the opposite direction from they-axis corresponds to the aforementioned orientation of the delineator Psought by calculation.

The combined area selector 29 of the image processing apparatus 12determines whether the position and the orientation of the object in thesecond image match the position and the orientation sought bycalculation (step S32). If the position and orientation of the object inthe second image do not match the position and orientation sought bycalculation (step S32: No), the combined area selector 29 performs areasetting and perspective conversion on the first image or the secondimage, thereby performing correction so that these images match (stepS34).

Step S32 is described using the example in FIG. 6. Since the positionPx(t1) of the object in the captured image of the left-side area Ci doesnot match P(t1) sought by calculation, the combined area selector 29performs correction so that these positions match. The combined areaselector 29 performs correction by performing area setting andperspective conversion on the first image or the second image. In thisexample, the movement direction of the object in the captured image ofthe left-side area Ci is not parallel to the y-axis, and therefore thecombined area selector 29 corrects the captured image of the left-sidearea Ci. In other words, the combined area selector 29 shifts thecaptured image of the left-side area Ci and performs perspectiveconversion so that the position Px(t1) of the object becomes theposition P(t1) sought by calculation.

After step S34, the combined area selector 29 again determines whetherthe position and the orientation of the object in the second image matchthe position and the orientation sought by calculation (step S32). Whenthe position and orientation match (step S32: Yes), the combined areaselector 29 combines the first image and the second image (step S38).

Here, in step S26, when the same object does not appear in the firstimage and the second image (step S26: No), the combined area selector 29terminates without performing correction. At this time, the first imageand the second image may be acquired again while the time differencebetween the captured times of the first image and the second image isadjusted.

As described above, the image processing apparatus 12 according to thisembodiment, which implements the aforementioned image processing method,not only achieves the same effects as Embodiment 1 but may also beapplied when the same object does not appear simultaneously in adjacentcaptured images.

Although this disclosure is based on embodiments and drawings, it is tobe noted that various changes and modifications will be apparent tothose skilled in the art based on this disclosure. Therefore, suchchanges and modifications are to be understood as included within thescope of this disclosure. For example, the functions and the likeincluded in the various components and steps may be reordered in anylogically consistent way. Furthermore, components or steps may becombined into one or divided.

For example, in Embodiment 1, the case of the vehicle 15 being in motionwas described, but even when the vehicle 15 is stopped, the same imageprocessing apparatus 12 may operate using the same object located insurrounding areas and appearing in two adjacent captured images. Forexample, suppose that the vehicle 15 is an automobile to which thecamera system 10 is attached, and that the vehicle 15 is conveyed by abelt conveyor. In this case as well, the same image processing apparatus12 as in Embodiment 1 may operate using a white line or the like paintedon the belt conveyor. In other words, automatic calibration that adjuststhe area setting and perspective conversion setting of the capturedimages is possible.

In Embodiment 1, the image processing apparatus 12 executes a sequenceof processing using objects that happen to exist outside the vehicle 15,but graphics drawn by laser light or the like irradiated from thevehicle 15 may be used as the aforementioned objects. In this case, theimage processing apparatus 12 can perform automatic calibration thatadjusts the area setting and perspective conversion setting of thecaptured images at any time.

For example, suppose that the vehicle 15 draws a ring-shaped graphic onthe road with laser light, and that the graphic overlaps the border B3,as illustrated in FIG. 8. In this case, for example with the sameprocessing as in Embodiment 1, the image processing apparatus 12 canadjust the area setting and perspective conversion setting of thecaptured images at any time. Unlike Embodiment 1, by the samenessdeterminer 28 knowing that a ring-shaped graphic is used for calibration(referred to below as prior knowledge) even though this graphic is not along, linear object, more effective processing can be executed.

In the example in FIG. 8, the graphic Ra in the captured image of thefront area Ai and the graphic Rc in the captured image of the left-sidearea Ci are discontinuous. The outline detector 27 performs outlinedetection on each of the captured image of the front area Ai (firstimage) and the captured image of the left-side area Ci (second image) togenerate the first outline image that includes the outline of thegraphic Ra and the second outline image that includes the outline of thegraphic Rc, the graphics Ra and Rc being portions of a ring-shapedgraphic. Based on the aforementioned prior knowledge, the samenessdeterminer 28 determines whether the first outline and the secondoutline are outlines of continuous objects (the ring-shaped graphic forcalibration). The combined area selector 29 performs area setting andperspective conversion on the captured image of the left-side area Ci sothat the graphic Rc is displayed at the position of the graphic Rc0 inFIG. 8 so as to be continuous at the border B3. In this way, automaticcalibration can be performed as a result of a particular graphic beingemitted from the vehicle 15. Such a graphic may, for example, be drawnby laser light emitted from the headlights of the vehicle 15.

A portion of the constituent elements of the camera system 10 accordingto the above-described embodiment may be provided external to thevehicle 15. For example, the image pickup apparatus, such as the frontcamera 11 a; the image processing apparatus 12; and the like may beimplemented as a communication device, such as a mobile phone or anexternal server, and be connected to the other constituent elements ofthe camera system 10 by a wired or wireless connection.

REFERENCE SIGNS LIST

-   -   10 Camera system    -   11 a Front camera    -   11 b Rear camera    -   11 c Left-side camera    -   11 d Right-side camera    -   12 Image processing apparatus    -   13 Display apparatus    -   14 Network    -   15 Vehicle    -   16 Door mirror    -   17 a, 17 b, 17 c, 17 d Optical system    -   18 a, 18 b, 18 c, 18 d Image pickup element    -   19 a, 19 b, 19 c, 19 d Image processor    -   20 a, 20 b, 20 c, 20 d Camera controller    -   21 Image acquisition unit    -   22 Control information acquisition unit    -   27 Outline detector    -   28 Sameness determiner    -   29 Combined area selector    -   Ai Front area    -   Bi Rear area    -   Ci Left-side area    -   Di Right-side area    -   B0, B1, B2, B3 Border    -   P Delineator    -   WL White line    -   WR White line

The invention claimed is:
 1. An image processing apparatus comprising:one or more processors configured to implement: an image acquisitionunit configured to acquire a first image capturing a surrounding area ofan automobile including a first area and a second image capturing asurrounding area of the automobile including a second area adjacent tothe first area; an outline detector configured to perform outlinedetection on the first image to generate a first outline of an objectwithin the first image and perform outline detection on the second imageto generate a second outline of an object within the second image,wherein at least one of the object within the first image and the objectwithin the second image extends across a border between the first imageand the second image; a sameness determiner configured to determine,based on the first outline and the second outline, whether the objectwithin the first image and the object within the second image is thesame object; and a combined area selector configured to determine, whenthe object within the first image and the object within the second imagehave been determined to be the same object, whether the first outlineand the second outline are continuous or discontinuous across the borderbetween the first image and the second image, wherein, when the combinedarea selector determines that the first outline and the second outlineare discontinuous across the border between the first image and thesecond image, the combined area selector performs area setting andperspective conversion on the first image or the second image to correcta discontinuity between the first outline and the second outline, andwherein the combined area selector is further configured to combine atleast the first image and the second image to generate a combined image,wherein, in the combined image, the first outline and the second outlineare continuous across the border between the first image and the secondimage.
 2. The image processing apparatus of claim 1, wherein the objectwithin the first image or the object within the second image is any oneselected from the group consisting of a display on a road surface, aguardrail, a side ditch, a road shoulder, and a delineator.
 3. The imageprocessing apparatus of claim 1, wherein the object within the firstimage or the object within the second image is a graphic drawn on a roadsurface by the automobile irradiating light.
 4. An image processingapparatus comprising: one or more processors configured to implement: animage acquisition unit configured to acquire a first image capturing asurrounding area of an automobile including a first area and a secondimage capturing a surrounding area of the automobile including a secondarea adjacent to the first area; an outline detector configured toperform outline detection on the first image to generate a first outlineof an object within the first image and perform outline detection on thesecond image to generate a second outline of an object within the secondimage; and a combined area selector configured to determine, based onthe first outline and the second outline and movement of the automobile,whether the object within the first image and the object within thesecond image is the same object, wherein, when the object within thefirst image and the object within the second image have been determinedto be the same object, the combined area selector determines a positionand an orientation of the object within the second image based on thefirst image and the movement of the automobile, performs area settingand perspective conversion on the second image, and combines at leastthe first image and the second image, in which area setting andperspective correction has been performed, to generate a combined image,and wherein, in the combined image, the position and orientation of theobject within the second image matches a position and an orientation ofthe object within the first image and corresponds to the movement of theautomobile.
 5. The image processing apparatus of claim 4, wherein thecombined area selector uses a movement direction and a speed of theautomobile.
 6. The image processing apparatus of claim 5, wherein thecombined area selector performs the area setting and the perspectiveconversion on the second image when the automobile is travelingstraight.
 7. The image processing apparatus of claim 4, wherein theobject within the first image or the object within the second image isany one selected from the group consisting of a display on a roadsurface, a guardrail, a side ditch, a road shoulder, and a delineator.8. An image processing method comprising: acquiring a first imagecapturing a surrounding area of an automobile including a first area anda second image capturing a surrounding area of the automobile includinga second area adjacent to the first area; performing outline detectionon the first image to generate a first outline of an object within thefirst image and performing outline detection to generate a secondoutline of an object within the second image, wherein at least one ofthe object within the first image and the object within the second imageextends across a border between the first image and the second image;sameness determining, based on the first outline and the second outline,whether the object within the first image and the object within thesecond image is the same object; determining, when the object within thefirst image and the object within the second image have been determinedto be the same object, whether the first outline and the second outlineare continuous or discontinuous across the border between the firstimage and the second image; when the first outline and the secondoutline are determined to be discontinuous across the border between thefirst image and the second image, performing area setting andperspective conversion on the first image or the second image to correcta discontinuity between the first outline and the second outline whenthe first outline and the second outline; and generating a combinedimage by combining at least the first image and the second image,wherein, in the combined image, the first outline and the second outlineare continuous across the border between the first image and the secondimage.
 9. An image processing method comprising: acquiring a first imagecapturing a surrounding area of an automobile including a first area anda second image capturing a surrounding area of the automobile includinga second area adjacent to the first area; performing outline detectionon the first image to generate a first outline of an object within thefirst image and performing outline detection on the second image togenerate a second outline of an object within the second image;determining, based on the first outline and the second outline andmovement of the automobile, whether the object within the first imageand the object within the second image is the same object; and when theobject within the first image and the object within the second imagehave been determined to be the same object, determining a position andan orientation of the object within the second image based on the firstimage and the movement of the automobile, performing area setting andperspective conversion on the second image, and combining at least thefirst image and the second image, in which area setting and perspectivecorrection has been performed, to generate a combined image, wherein, inthe combined image, the position and orientation of the object withinthe second image matches a position and an orientation of the objectwithin the first image and corresponds to the movement of theautomobile.